Unsupervised record matching with noisy and incomplete data

We consider the problem of duplicate detection: given a large data set in which each entry has multiple attributes, detect which distinct entries refer to the same real world entity. Our method consists of three main steps: creating a similarity score between entries, grouping entries together into ‘unique entities’, and refining the groups. We compare various methods for creating similarity scores, considering different combinations of string matching, term frequency-inverse document frequency methods, and n-gram techniques. In particular, we introduce a vectorized soft term frequency-inverse document frequency method, with an optional refinement step. We test our method on the Los Angeles Police Department Field Interview Card data set, the Cora Citation Matching data set, and two sets of restaurant review data. The results show that in certain Y. van Gennip University of Nottingham E-mail: [email protected] B. Hunter Claremont McKenna College E-mail: [email protected] A. Ma Claremont Graduate University E-mail: [email protected] D. Moyer University of Southern California E-mail: [email protected] R. de Vera formerly California State University, Long Beach E-mail: [email protected] A. L. Bertozzi University of California, Los Angeles E-mail: [email protected] parameter ranges soft term frequency-inverse document frequency methods can outperform the standard term frequency-inverse document frequency method; they also confirm that our method for automatically determining the number of groups typically works well in many cases and allows for accurate results in the absence of a priori knowledge of the number of unique entities in the data set.